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Amendments to the Claims : 
This Usting of claims replaces all prior versions and listings of claims in the application: 

Listing of Claims : 

1-24. (cancelled) 

25. (Previously presented) The method of claim 29, wherein the surface comprises a two 

dimensional surface. 

26-28. (Cancelled) 

29. (Currently Amended) A method comprising: 

obtaining a three dimensional model of a three dimensional environment, the three 
dimensional model generated from range sensor information representing a heigjit field for the 
three dimensional environment; 

identifying in real time a region in motion with respect to a background image in real- 
time video imagery information from at least one image sensor having associated position and 
orientation information with respect to the three dimensional model, the background image 
comprising a single distribution background dynamically modeled from a time average of the 
real-time video imagery information; 

placing a surface that corresponds to the moving region in the three dimensional models 
wherein placing the surface comprises casting a ray from an optical center, correspondinR to the 
real-time video imagery information, to a bottom point of the moving region in an image plane in 
the three dimensional model and determining a position, an orientation and a size of the surface 
based on the ray, a ground plane in the three dimensional model, and the moving region : 
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projecting the real-time video imagery information onto the three dimensional model, 
including the surface, based on the position and orientation information; and 

visualizing the three dimensional model with the projected real-time video imagery; 

wherein identifying a region in motion in real time comprises subtracting the background 
image from the real-time video imagery information, identifying a foreground object in the 
subtracted real-time video imagery information, validating the foreground object by correlation 
matching between identified objects in neighboring image frames, and outputting the validated 
foreground object.[[;]] 

wh e r e in identifiing a foreground obj e ct oomprio e o id e ntifying th e foreground object in 
the subfraot e d r e al tim e vid e o imagery information using a histogram bas e d thr e shold and a 



wher e in identifying a region in motion in r e al time furth e r compris e s e stimating th e 
background imago by modeling th e background image as a t e mporal pix e l av e rag e of fiv e r e c e nt 
imag e frames in th e real time vid e o imagery information. 

30. (Previously presented) The method of claim 29, further comprising tracking the 
position and orientation information of the at least one image sensor in the environment with 
respect to the three dimensional model in real-time. 

3 1 . (Previously presented) The method of claim 30, wherein obtaining a three 
dimensional model of a three dimensional environment comprises generating the three 
dimensional model of the three dimensional environment. 

32. (Cancelled) 

33. (Previously presented) The system of claim 37, wherein the surface comprises a two 
dimensional surface. 



34-36. (Cancelled) 



Applicant : Ulrich Neumann et al. Attorney's Docket No.: 06666-156001 /#3345A 

Serial No. : 10/676,377 

Filed : September 30, 2003 

Page : 4 of 13 



37. (Currently Amended) An augmented virtual environment system comprising: 

an object detection and tracking component that identifies in real time a region in motion 
with respect to a background image in real-time video imagery information firom at least one 
image sensor having associated position and orientation information with respect to a three 
dimensional model of a three dimensional environment, the three dimensional model generated 
from range sensor information representing a height field for the three dimensional environment, 
the background image comprising a single distribution background dynamically modeled from a 
time average of the real-time video imagery information, and places a surface that corresponds to 
the moving region with respect to the three dimensional model , wherein the object detection and 
tracking component places the surface by performing operations comprising casting; a ray from 
an optical center, corresponding to the real-time video imagery infoiTnation, to a bottom point of 
the moving region in an image plane in the three dimensional model, and determining a position, 
an orientation and a size of the surface based on the rav. a ground plane in the three dimensional 
model, and the moving region : 

a dynamic fusion imagery projection component that projects the real-time video imagery 
information onto the three dimensional model, including the surface, based on the position and 
orientation information; and 

a visualization sub-system that visualizes the three dimensional model with the projected 
real-time video imagery; 

wherein the object detection and tracking component identifies the moving region by 
performing operations comprising subtracting the background image firom the real-time video 
imagery information, identifying a foreground object in the subtracted real-time video imagery 
information, validating the foreground object by correlation matching between identified objects 
in neighboring image frames, and outputting the validated foreground object.[[;]] 

wherein identifying a foreground object. Gomprises identifying the foreground object in 
the subtracted real time video imag e ry information using a histogram bas e d thr e shold and a 
nois e filter; and 
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wherein identifiing a region in motion in real time furthor compriseo e stimating tho 
background imago by modeling th e background imago as a t e mporal pix e l averag e of fivo rooont 
imag e fram e s in th e r e al time video imag e ry inform atieHr 

38. (Previously presented) The system of claim 37, further comprising a tracking sensor 

system that integrates visual input, global navigational satellite system receiver input, and inertial 
orientation sensor input to obtain the position and the orientation information associated with the 
at least one image sensor in real time in conjunction with the real-time video imagery. 

39. (Previously presented) The system of claim 38, further comprising a model 
construction component that generates the three dimensional model of the three dimensional 
environment. 

40-49. (Cancelled) 

41. (New) The method of claim 29, wherein identifying a foreground object comprises 
identifying the foreground object in the subtracted real-time video imagery information using a 
histogram-based threshold and a noise filter. 

42. (New) The method of claim 41, wherein identifying a region in motion in real time 
further comprises estimating the background image by modeling the background image as a 
temporal pixel average of five recent image firames in the real-time video imagery information. 

43. (New) The system of claim 37, wherein identifying a foreground object comprises 
identifying the foreground object in the subtracted real-time video imagery information using a 
histogram-based threshold and a noise filter. 
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44. (New) The system of claim 43, wherein identifying a region in motion in real time 
further comprises estimating the background image by modeling the background image as a 
temporal pixel average of five recent image frames in the real-time video imagery information. 

45. (New) A machine-readable medium embodying information indicative of instructions 
for causing one or more machines to perform operations comprising: 

obtaining a three dimensional model of a three dimensional environment, the three 
dimensional model generated from range sensor information representing a height field for the 
three dimensional environment; 

identifying in real time a region in motion with respect to a background image in real- 
time video imagery information from at least one image sensor having associated position and 
orientation information with respect to the three dimensional model, the background image 
comprising a single distribution background dynamically modeled from a time average of the 
real-time video imagery information; 

placing a surface that corresponds to the moving region in the three dimensional model, 
wherein placing the surface comprises casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the moving region in an image plane in 
the three dimensional model, and determining a position, an orientation and a size of the surface 
based on the ray, a ground plane in the three dimensional model, and the moving region; 

projecting the real-time video imagery information onto the three dimensional model, 
including the surface, based on the position and orientation information; and 

visualizing the three dimensional model with the projected real-time video imagery; 

wherein identifying a region in motion in real time comprises subtracting the backgroimd 
image from the real-time video imagery information, identifying a foreground object in the 
subtracted real-time video imagery information, validating the foreground object by correlation 
matching between identified objects in neighboring image frames, and outputting the validated 
foreground object. 
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46. (New) The machine-readable medium of claim 45, wherein the surface comprises a 

two dimensional surface. 

47. (New) The machine-readable medium of claim 45, further comprising tracking the 
position and orientation information of the at least one image sensor in the environment with 
respect to the three dimensional model in real-time. 

48. (New) The machine-readable medium of claim 47, wherein obtaining a three 
dimensional model of a three dimensional environment comprises generating the three 
dimensional model of the three dimensional environment. 

49. (New) The machine-readable medium of claim 45, wherein identifying a foreground 
object comprises identifying the foreground object in the subtracted real-time video imagery 
information using a histogram-based threshold and a noise filter. 

50. (New) The machine-readable medium of claim 45, wherein identifying a region in 
motion in real time further comprises estimating the backgroimd image by modeling the 
background image as a temporal pixel average of five recent image fi-ames in the real-time video 
imagery information. 



